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AR!  Research  Reports  and  Technical  Reports  are  intended  for  sponsors  of 
R&O  tasks  and  for  other  research  and  military  agencies.  Any  findings  ready 
for  implementation  at  the  time  of  publication  are  presented  in  the  last  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recom¬ 
mendations  for  official  action  normally  are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


FOREWORD 


The  principal  mission  of  the  Training  Research  Laboratory  of  the  Army  Re¬ 
search  Institute  is  to  refine  the  Army's  training  methods  and  practices.  As 
part  of  this  refinement,  the  mission  of  ARI-Fort  Knox  and  the  Training  Technol¬ 
ogy  Field  Activity  (TTFA)  is  to  develop,  evaluate,  and  Implement  new  technol¬ 
ogies  and  methods  into  Army  training.  The  TTFA's  recent  efforts  have  focused 
on  the  potential  of  computer  and  videodisc-based  instruction  (CVBI)  for  Armor 
training.  This  research  effort  evaluated  the  training  efficiency  and  effective¬ 
ness  of  CVBI  courseware  developed  by  the  TTFA  to  train  the  remedial  skills  pre¬ 
requisite  to  the  Basic  Noncommissioned  Officer  Course  (BNCOC)  for  Ml  tank 
commanders . 

This  effort  is  part  of  the  ARI-Fort  Knox  Field  Unit's  research  program  to 
apply  new  training  technology  to  Armor  skills  training.  A  Memorandum  of  Agree¬ 
ment  covering  the  application  of  training  technology  to  Armor  skills  training, 
"Establishment  of  Field  Training  Technology  Activity,  Fort  Knox,  Kentucky,”  was 
signed  by  the  U.S.  Army  Training  and  Doctrine  Command  (TRADOC),  the  U.S.  Army 
Armor  Center  (USAARMC),  and  the  U.S.  Army  Research  Institute  (ARI)  on  4  November 
1983.  Results  from  this  evaluation  of  CVBI  remedial  courseware  were  provided  to 
the  Chief  of  the  Training  Technology  Field  Office  at  Fort  Knox  and  briefed  to 
the  Technical  Director  of  the  U.S.  Army  Armor  School  and  TRADOC s  Deputy  Chief 
of  Staff  for  Training.  Successful  implementation  of  this  courseware  into  BNCOC 
would  support  the  transfer  of  this  courseware  to  other  Army  training  systems. 
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REMEDIAL  SKILLS  TRAINING:  AN  EVALUATION  OF  COMPUTER  AND 
VIDEODISC-BASED  COURSEWARE 


EXECUTIVE  SUMMARY 


Requirement : 

To  evaluate  the  effectiveness  of  computer  and  vldeodlsc-based  courseware 
for  training  remedial  skills  to  Ml  tank  commanders. 


Procedure : 

As  part  of  the  Training  Technology  Field  Activity  (TTFA)  mission  to  pro¬ 
vide  innovative  training  technologies  for  Army  training,  computer  and  video¬ 
disc-based  instruction  (CVBI)  was  developed  for  training  a  set  of  remedial 
tasks  that  are  prerequisite  skills  for  admission  into  the  19K  Basic  Noncommis¬ 
sioned  Officer  Course  (BNOCC)  for  Ml  tank  commanders.  The  training  effective¬ 
ness  of  this  CVBI  was  evaluated  with  control  (ii  =  44)  and  experimental  = 

39)  groups  of  military  personnel  similar  to  soldiers  in  19K  BNCOC.  All  train¬ 
ing  and  tests  were  on-line  products  provided  by  a  CVBI  delivery  system.  A 
pretest  versus  posttest  design  with  several  control  conditions  were  used  to 
determine  the  CVBI's  training  effectiveness. 


Findings : 

Results  for  the  four  remedial  tasks  included  in  this  evaluation — Use  Visual 
Signals,  Establish  Tank  Firing  Positions,  Determine  Grid  Coordinates,  and  Iden¬ 
tify  Armored  Vehicles — showed  significant  improvements  in  posttest  performance. 
In  addition,  participants'  subjective  evaluations  of  this  CVBI  clearly  indicated 
they  prefer  CVBI  over  conventional  training  methods.  While  training  efficiency 
could  not  be  directly  determined,  the  diversity  of  participants'  need  for  reme¬ 
dial  training  strongly  supports  the  adaptive  nature  of  training  technologies 
such  as  CVBI  in  the  Army  training  system. 


Utilization  of  Findings: 

The  findings  of  this  evaluation  were  provided  to  the  Technical  Director  of 
the  Armor  School  who  recommended  validation  of  this  courseware  in  the  19K  BNCOC 
Program  of  Instruction.  To  facilitate  the  transfer  of  this  CVBI,  the  courseware 
should  be  revised  to  meet  the  functional  specifications  of  the  Electronic  Infor¬ 
mation  Delivery  System  (EIDS).  As  the  designated  Army  standard  for  CVBI,  BIDS 
has  further  established  the  Army's  commitment  to  incorporating  these  TTFA  train¬ 
ing  technologies  into  the  Army  training  system. 
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REMEDIAL  SKILLS  TRAINING:  AN  EVALUATION  OF  COMPUTER  AND 
VIDEODISC-BASED  COURSEWARE 


INTRODUCTION 

While  the  Army  training  system  has  incorporated  numerous  new  technologies 
in  recent  years,  it  is  still  heavily  dependent  upon  approaches  from  an  ear¬ 
lier  era,  such  as  use  of  paper  materials  and  classroom  lecturing  techniques. 
In  an  operational  training  environment  it  is  difficult  to  test  and  apply 
promising  technologies  and  innovations,  especially  given  the  rapid  rate  at 
which  technologies  are  being  developed  today. 

To  overcome  this  difficulty  the  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC),  the  U.S.  Army  Armor  Center  (USAARMC),  and  the  U.S.  Army  Research 
Institute  (ARI)  have  established  a  Training  Technology  Field  Activity  (TTFA) 
at  Fort  Knox,  Kentucky.  The  mission  of  the  TTFA  is  to  systematically  iden¬ 
tify,  evaluate,  and  Introduce  new  training  methods,  techniques,  technologies, 
and  models  to  Army  training.  A  more  complete  discussion  of  TTFA's  back¬ 
ground,  approach  and  scope  of  effort  is  provided  by  Kristiansen  (1986). 

The  current  efforts  of  the  Fort  Knox  TTFA  have  focused  on  the  Basic  Non¬ 
commissioned  Officer  Course  for  Ml  tank  commanders,  19K  BNCOC.  TTFA's  ini¬ 
tial  projects  have  attempted  to  develop  and  Implement  computer  and 
videodisc-based  (CVBI)  training  technologies.  For  these  initial  projects  two 
blocks  of  Instruction,  clusters  of  tasks,  were  selected  from  the  BNCOC  Pro¬ 
gram  of  Instruction  (POI)  for  conversion  to  CVBI.  The  first  block,  land 
navigation,  included  training  and  testing  for  five  common  military  tasks 
regarded  as  fundamental  to  daytime  land  navigation.  CVBI  courseware  for 
these  tasks  was  developed  (Elder,  Harris,  Sticha,  Stein,  Rnerr,  &  Tkacz, 
1985),  and  its  training  effectiveness  and  efficiency  was  evaluated  at  Fort 
Knox  (Llckteig  &  Burnside,  1986).  For  the  second  block  of  Instruction,  reme¬ 
dial  training,  CVBI  courseware  was  developed  for  a  set  of  prerequisite  skills 
that  are  required  for  admission  into  the  19K  BNCOC  course.  This  paper  re¬ 
ports  on  the  evaluation  of  this  remedial  courseware's  training  effectiveness. 


BACKGROUND 

As  recommended  by  the  Defense  Science  Board's  1982  Summer  Study  on  Train¬ 
ing  and  Training  Technology,  the  TTFA  initiated  a  program  of  research  and 
development  to  explore  the  design,  application  and  effectiveness  of  innova¬ 
tive  training  technologies.  Once  19K  BNCOC  was  selected  as  the  initial  focus 
of  these  training  technologies,  the  TTFA  ensured  that  >his  course's  redesign 
and  development  was  guided  by  the  Systems  Approach  to  Training  (SAT)  as  de¬ 
tailed  in  TRADOC  Reg  350-7  (1982).  Consistent  with  the  Instructional  System 
Development  (ISD)  model,  this  course  development  was  based  on  a  thorough 
analysis  of  the  training  requirements  for  19K  BNCOC  (Drucker,  Hannaman, 
Melching,  &  O'Brien,  1985).  A  primary  goal  of  this  analysis  was  to  identify 
those  tank  commander  tasks  critical  for  mission  accomplishment  and  Individual 
survival  on  the  battlefield.  This  analysis  also  identified  a  set  of  tasks 
that  were  prerequisite  requirements,  skill  level  (SL)  1  and  2,  to  the  SL  3 
tasks  equivalent  to  the  19K30  course  classification. 


1 


In  a  subsequent  phase  of  this  SAT  effort  training  systems  and  technol¬ 
ogies  were  reviewed  and  analyzed  for  their  applicability  to  the  19K  BNCOC  POl 
(Drucker  et  al.,  1985).  A  total  of  27  different  training  systems  were  Iden¬ 
tified  for  consideration  by  BNCOC  training  developers,  and  the  potential 
Implications  of  implementing  each  of  these  training  systems  Into  the  BNCOC 
course  were  analyzed  with  respect  to  the  course's  training  developers,  In¬ 
structors  and  students.  As  for  training  technologies,  the  SAT  analysis  noted 
the  Army's  Increasing  use  of  automated  training  technologies  and  in  particu¬ 
lar  computer-based  Instruction  (CBI)  and  computer-managed  instruction  (CMI). 
One  of  the  most  Innovative  technologies  Identified  In  this  analysis  was  In¬ 
teractive  videodisc  which  combines  the  high  fidelity  Imagery  of  Instructional 
television  with  the  powerful  Instructional  features  of  CBI. 

The  SAT  analysis  In  conjunction  with  an  SAT  design  for  the  BNCOC  course 
(Morrison,  Drucker,  &  O'Brien,  1985)  ensured  that  the  TTFA's  efforts  In  19K 
BNCOC  were  guided  by,  and  In  accordance  with,  the  Initial  phases  of  the  ISD 
model  for  course  development  and  revision. 

With  this  background  the  TTFA  then  moved  to  the  next  ISD  phase,  course¬ 
ware  development.  The  TTFA  effort  for  19K  BNCOC  entailed  courseware  develop¬ 
ment  efforts  for  most  of  the  subject  or  content  areas  Included  In  the 
classroom  portion  of  the  POX  such  as  land  navigation,  communications,  fire 
commands,  call  for  and  adjust  Indirect  fire  as  well  as  nuclear,  biological 
and  chemical  (NBC)  defense.  This  report,  however.  Is  limited  to  the  evalua¬ 
tion  of  one  additional  block  of  BNCOC  courseware,  remedial  training,  and  will 
begin  with  a  brief  discussion  of  this  particular  courseware's  design  and 
development. 


REMEDIAL  COURSEWARE  DESIGN  AND  DEVELOPMENT 


Courseware  Content 


As  the  TTFA's  focus  shifted  from  the  overall  analysis  and  design  of  the 
BNCOC  course  to  the  remedial  training  requirements  for  tank  commanders,  can¬ 
didate  tasks  for  prerequisite  training  were  thoroughly  reviewed  (Knerr, 

Elder,  Campbell,  Stein,  Stlcha,  &  Morrison,  1986).  This  analysis  Included 
the  previously  noted  prerequisite  tasks  identified  during  the  overall  SAT 
analysis  (Drucker  et  al.,  1985),  the  tasks  currently  included  in  the  BNCOC 
diagnostic  tests,  and  prerequisite  tasks  listed  In  the  Soldier's  Manuals 
FM  17-19K  1/2/3.  In  addition,  this  preliminary  analysis  of  remedial  tasks 
included  Interviews  with  Instructors  and  course  managers,  review  of  the  stu¬ 
dent  performance  records  on  diagnostic  tests,  and  guidance  by  a  steering 
committee  of  subject  matter  experts. 

The  original  pool  of  70  remedial  tasks  Identified  as  potential  candidates 
for  the  remedial  courseware  was  reduced  to  21  by  eliminating  tasks  with  pre¬ 
viously  high  pass  rates,  tasks  not  directly  related  to  SL  3  tasks  Included  In 
the  POI,  and  tasks  recommended  for  the  requisite  POI  by  the  SAT  design  effort 
(Morrison  et  al.,  1985).  From  these  21  tasks,  the  following  set  of  5  prereq¬ 
uisite  tasks  for  remedial  courseware  development  were  selected  on  the  basis 
of  their  testability  and  suitability  for  CBI: 
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o  Determine  grid  coordinates  of  a  point  on  a  military  map  using  the 
military  grid  reference  system  (Task  071-329-1002). 

o  Communicate  using  visual  signalling  techniques  (Task 
071-326-0608). 

o  Recognize  and  Identify  friendly  and  threat  armored  vehicles  (Task 
878-920-1001). 

o  Establish  tank  firing  positions  (Task  171-123-1008). 

o  Operate  a  radio  set  (Task  113-587-2043). 

These  five  SL  1  tasks  were  then  subjected  to  standard  and  cognitive  task 
analyses  to:  identify  common  errors,  determine  the  relative  difficulty  of 
learning  the  tasks,  and  infer  the  cognitive  structures  underlying  their  skill 
acquisition  and  performance.  Training  strategies  and  learning  principles  for 
each  of  these  tasks  were  then  derived  by  application  of  the  Training  Effec¬ 
tiveness  and  Cost  Prediction  (TECEP)  method.  For  a  more  complete  discussion 
of  these  remedial  courseware  analysis  and  design  phases  the  reader  Is  re¬ 
ferred  to  Knerr  et  al.  (1986). 


Courseware  Delivery:  Training  Technologies 

As  part  of  TTFA's  mission  to  Implement  innovative  training  technologies 
Into  Army  training,  the  land  navigation  courseware  was  designed  and  developed 
as  computer-based  instruction  (CBI).  The  positive  instructional  characteris¬ 
tics  associated  with  CBI  such  as  reliability,  standardization,  cost  effec¬ 
tiveness,  Interactivity  and  Immediate  feedback  have  been  well  articulated 
(e.g.,  Ellis,  1986;  Helnich,  1985;  Orlansky  &  String,  1981).  A  more  Innova¬ 
tive  technology  included  In  the  courseware  design  was  the  use  of  videodisc 
Imagery  for  depicting  live-action  modeling  of  task  performance  and  dynamic 
real-world  terrain  scenes. ’  Interactive  video  provides  a  number  of  unique 
features  such  as  fidelity  of  visual  and  audio  training  materials,  random  and 
rapid  access,  multiple  speeds  and  directions,  and  the  potential  for  Interac¬ 
tive  and  Individualized  Instruction.  A  more  complete  discussion  of  the  mer¬ 
its  of  Interactive  video,  beyond  the  current  scope,  is  available  (Hannafln, 
1985;  Relgeluth  &  Garfield,  1984). 

With  respect  to  the  current  TTFA  Implementation,  Instructional  experts 
suggest  that  Interactive  video  may  be  particularly  effective  In  applied 
training  settings  and  In  an  effort  to  solve  troublesome  training  issues  not 
resolved  by  conventional  training  methods  (Woolley,  1982)  In  summary,  an 
Interactive,  multimedia  training  package  was  developed  by  Interfacing  the 
computer-based  instructional  materials,  text  and  graphics,  with  supporting 
vldeodlsc-based  materials,  both  video  and  audio. 

The  Fort  Knox  TTFA  effort  utilized  the  MlcroTICCIT  System  II  developed  by 
the  Hazeltlne  Corporation.  Time-Shared  Interactive  Computer-Controlled  In¬ 
formation  Television  (TICCIT)  is  an  Integrated  hardware,  software  and  course¬ 
ware  development  system  designed  for  the  efficient  production,  delivery  and 
management  of  computer-based  Instruction.  This  system  Is  not  only  tailored 
to  meet  the  training  developer's  needs  for  courseware  authoring  and  delivery. 
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but  also  for  computer  managed  instruction  (CMI)  in  which  student  records  and 
progress  are  automaf ically  monitored  and  maintained.  System  II  was  designed 
for  mid-scale  (i.e.  40  workstations)  CBI  Implementations,  consistent  with 

the  current  and  projected  19K  BNCOC  roster  size.  MicroTICCIT  was  also  se¬ 
lected  because  of  its  ability  to  integrate  peripheral  instructional  technolo¬ 
gies  such  as  the  interactive  videodisc  used  for  this  remedial  courseware,  and 
speech  synthesis  and  recognition  systems  used  in  other  TTFA  courseware  devel¬ 
opments  for  BNCOC. 


Courseware  Structure 


The  remedial  courseware  structure  was  based  on  MicroTICCIT* s  ADAPT,  Level 
3,  authoring  language  and  courseware  structure.  ADAPT  provides  four  levels 
of  instruction:  course,  unit,  lesson  and  segment  (CULS);  and  an  underlying 
map  hlerachy  to  process  the  sequencing  of  instructional  materials.  Course¬ 
ware  for  each  of  the  five  remedial  tasks  was  designed  as  a  separate  lesson 
within  the  Remedial  Unit,  and  under  the  19K  BNCOC  Course.  All  training  and 
tests  for  each  of  the  five  remedial  tasks  were  on-line  CBI  products.  Each  of 
these  five  lessons,  corresponding  to  the  five  remedial  tasks,  contained  a 
number  of  segments  which  with  minor  exceptions  were  as  follows: 

o  Objective 

o  Instruction 

-  Review 

-  Help 

o  Practice 

-  Feedback 

-  Help 

o  Test 

These  segments  were  presented  to  the  students  through  a  hlerarchlal  series  of 
menus.  All  student  requests  from  these  menus  and  all  student  Inputs  and 
responses  were  entered  by  means  of  a  llghtpen.  The  segments  are  briefly 
described  in  the  following  paragraphs. 

The  Objective  presented  the  overall  training  goal,  terminal  objective,  of 
the  lesson  and  Included  the  conditions,  actions  and  standards  associated  with 
the  task.  The  Instruction  segment  provided  the  actual  training  for  the  task 
and  Integrated  text,  video,  audio  and  graphic  Instructional  materials.  In 
support  of  this  instructional  segment,  the  Review  component  provided  students 
an  opportunity  to  quickly  access  Informative  summaries  of  the  complete  in¬ 
structional  package,  and  the  Help  component  provided  additional  explanatory 
information  specific  to  key  issues  addressed  by  the  instructional  segment. 

The  Practice  segment  provided  students  the  opportunity  to  pretest  their  mas¬ 
tery  of  the  instructional  requirements  with  sample  items  similar  to  those 
included  in  the  final  test.  Students  responses  were  evaluated  by  the  Feed¬ 
back  component  which  provided  them  immediate  knowledge  of  their  results  for 
individual  practice  items,  and  assessed  their  readiness  to  advance  to  the 
Test  segment. 
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The  test  segment  consisted  of  performance-oriented  knowledge  tests  which 
have  also  been  described  as  synthetic  tests  (Osborn,  1970;  Osborn  and  Ford, 
1978).  The  Intent  of  performance-oriented  knowledge  tests  Is  to  provide  a  re¬ 
liable  and  valid  Indicator  of  student  mastery  In  the  absence  of  hands-on 
performance  evaluation  measures.  Key  features  of  performance-oriented  knowl¬ 
edge  test  are  that  task  stimulus  conditions  are  simulated,  and  facslmllles  of 
actual  task  responses  are  required  for  test  performance.  Performance- 
oriented  stimulus  and  response  conditions  for  this  courseware  are  described 
In  the  subsequent  Courseware  Instruction  and  Evaluation  Measures  sections.  A 
comparison  of  these  remedial  performance-oriented  knowledge  tests  with  con¬ 
ventional  hands-on  measures  was  conducted  and  Is  provided  by  Knerr  et  al. 
(1986). 


Courseware  Instruction 


The  courseware  was  designed  to  maximize  the  training  potential  of  the 
Innovative  technologies  Included  In  this  TTFA  effort.  In  particular,  this 
CVBI  courseware  attempted  to  provide  a  multimedia  training  package  responsive 
to  remedial  and  Individual  training  requirements.  The  following  section 
provides  a  brief  description  of  this  CVBI  training  for  each  of  the  five  reme¬ 
dial  tasks  and  the  manner  In  which  these  training  technologies  were  Inte¬ 
grated  Into  the  remedial  courseware  package  for  19K  BNCOC. 

Determine  Grid  Coordinates.  After  a  review  of  the  lesson's  objective, 
the  Instruction  for  this  task  proceeded  to  a  videodisc-based  live-action 
model  of  a  soldier  In  a  typical  field  setting  demonstrating  and  explaining 
how  to  determine  grid  coordinates.  Text  and  graphic  messages  were  used  to 
reinforce  these  procedures  and  emphasize  the  following  two  task  actions: 

(1)  determine  the  6-dlglt  coordinates  for  points  marked  on  the  map,  and 

(2)  mark  the  point  on  the  map  corresponding  to  the  6-dlglt  coordinates  desig¬ 
nated.  Designated  coordinates  and  locations  appeared  on  the  student's 
workstation  monitor  which  also  displayed  a  standard  1:30,000  military  map, 
marginal  map  Information,  and  a  graphic  overlay  of  a  numerical  key  pad  for 
Inputting  each  student's  coordinate  responses. 

Use  Visual  Signals.  This  Instructional  segment  began  with  a  live  action 
model  of  a  soldier  explaining  why  visual  signals  were  important  and  demon¬ 
strating  various  examples  of  the  44  arm-and-hand,  flag  and  light  signals 
Included  In  the  lesson.  Examples  were  reinforced  with  helpful  hints  for 
remembering  each  signal.  Signals  normally  given  by  a  tank  commander  were 
filmed  showing  the  commander  signalling  from  the  commander's  hatch  on  the 
tank.  Signals  normally  received  by  the  tank  commander  were  filmed  as  viewed 
by  the  commander.  During  the  practice  Items  students'  were  required  to  Iden¬ 
tify  either  signals  received,  or  to  select,  from  a  split-screen  presentation 
of  four  visual  messages,  signals  to  be  given. 

Identify  Armored  Vehicles.  The  Instruction  began  with  a  narrator  asking 
student's  to  Imagine  they  were  In  a  tactical  situation  and  in  a  turret-down 
position.  The  narration  emphasized  the  need  for  Immediate  and  accurate  Iden¬ 
tification  given  the  combat  scenario.  Students  were  then  presented  with  full 
color,  scale-model  Images  of  30  friendly  and  threat  armored  vehicles  as 
viewed  from  five  different  perspectives.  Key  recognition  features  and  the 
armament  of  each  vehicle  were  highlighted  by  graphic  arrows,  and  the  correct 
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nomenclature  (l.e.,  weapon  system  model)  was  provided.  Practice  Items  first 
required  students  to  Indicate  whether  the  vehicle  was  a  friendly  or  threat 
weapon  system  and  then  to  select,  from  a  multiple-choice  list,  the  correct 
nomenclature.  Students  were  allowed  10  seconds  for  each  of  these  responses. 

Establish  Tank  Firing  Posltons.  This  lesson  began  with  a  discussion  and 
graphic  presentation  of  tanks  In  various  turret-dovm,  hull-down  and  hide 
positions  In  a  field  setting.  Graphic  aids  were  used  to  Illustrate  the  ob¬ 
servation  and  fields  of  fire  available  from  each  position,  and  the  relative 
coverage  and  concealment  provided  by  each  position.  During  the  practice 
Items  for  this  lesson  students  were  required  to  "move"  the  tank  Images  across 
the  monitor  and  Into  each  of  these  positions  as  designated.  This  Instruction 
also  Included  a  discussion  and  Illustrations  of  the  primary,  alternate  and 
supplementary  tank  firing  positions  used  In  the  defensive  battle  position. 
Students  were  required  to  designate  each  of  these  positions  for  various  com¬ 
bat  situations. 

Operate  a  Radio  Set.  The  instruction  for  this  lesson  was  based  on  a 
graphic  simulation  of  an  AN/VRC-64  radio  on  which  all  switches  and  dials 
could  be  set  or  adjusted  to  any  realistic  position  via  the  llghtpen.  The 
Instruction,  using  both  video  and  audio,  differentiated  the  various  dials  and 
settings  required  for  preparing,  operating  and  shutting  down  the  radio  sys¬ 
tem.  Setting  sequences  with  fixed  or  random  orders  were  emphasized.  In  addi¬ 
tion,  students  were  provided  a  hard-copy  of  the  Communications  Electronic 
Operations  Instructions  (CEOI)  for  determining  the  operating  frequency,  call 
signs  and  authentication  procedures.  For  practice  Items  students  were  re¬ 
quired  to  set  the  dials  and  switches  required  for  preparing,  operating  and 
shutting  down  the  system  and  to  Input  Information  as  required  from  the  CEOI. 

It  must  be  noted,  however,  that  during  the  pilot  phase  of  this  evaluation 
It  was  discovered  that  one  of  the  key  switches  on  the  simulated  radio  could 
not  be  properly  set.  Neither  In-house  nor  contractor  personnel  were  able  to 
identify  the  cause  of  this  problem  or  correct  it.  Since  this  problem  would 
prevent  students  from  successfully  completing  the  practice  and  test  Items, 
the  courseware  for  this  task  was  not  included  In  this  evaluation  and  will  not 
be  considered  in  the  following  sections.  Action  Is  underway  to  correct  this 
problem  so  that  the  courseware  can  be  tested  In  subsequent  evaluation  ef¬ 
forts. 


RESEARCH  OBJECTIVES 

The  primary  research  Issue  was  to  determine  the  training  effectiveness  of 
the  remedial  CVBI  courseware.  It  was  hypothesized  that' the  courseware  would 
significantly  Increase  soldiers'  posttest  scores,  above  their  pretest  scores, 
for  each  of  the  four  remedial  tasks  addressed  by  this  courseware.  Two  con¬ 
trol  conditions  were  Included  to  counter  the  alternative  hypotheses  that 
posttest  Increases  may  have  been  caused  by  soldiers'  Increased  familiarity 
with  the  pretest  Items  and/or  the  courseware  delivery  system,  due  to  each 
participants  approximately  three  hours  of  testing  and  training.  Additional 
research  issues  were  to  evaluate  the  courseware's  training  efficiency,  in 
terms  of  time  required  to  complete  the  training,  and  the  ease-of-use  soldiers 
experienced  with  the  courseware. 
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METHOD 


Participants 

Participants  for  this  research  were  E3-E5  personnel  stationed  at  Fort 
Knox  with  either  an  Armor  (MOS  19K)  or  a  Calvary  (MOS  19D)  military  bcck- 
ground.  They  were  selected  as  a  sample  representative  of  the  NCOs  that  might 
receive  this  courseware  In  the  formal  program  of  Instruction  (POI)  for  Ml 
tank  commanders,  19K  BNCOC.  Thirty-nine  soldiers,  the  remedial  group,  par¬ 
ticipated  In  the  formal  evaluation.  An  additional  44  soldiers,  the  land 
navigation  group,  served  as  a  control  group  for  this  evaluation.  Partici¬ 
pants  from  both  of  these  groups  averaged  just  over  three  years  of  military 
service,  with  a  range  of  one  to  eight  years.  They  also  averaged  23  years  of  - 
age,  with  a  range  of  19  to  30  years  of  age.  Almost  all  (93Z)  participants  had 
completed  high  school,  and  nearly  one  out  of  four  had  completed  some  college 
courses.  Most  of  them  reported  little  or  no  computer  experience.  Based  on 
an  Informal  comparison  with  biographic  data  of  19K  BMCOC  personnel,  both 
samples  appeared  representative  of  19K  BNCOC  attendees.  Due  to  the  formative 
aspects  of  this  evaluation  and  the  limited  number  of  student  workstations 
available.  It  was  decided  cot  to  disrupt  the  BNCOC  course  cycle  by  using 
students  currently  enrolled  In  19K  BNCOC. 


Apparatus 

The  CBI  for  this  evaluation  was  supported  by  a  MlcroTICCIT  (System  II) 
host  computer  networked  to  the  student  workstations.  Each  workstation  in¬ 
cluded  the  following  hardware: 

o  IBM  PC  256K  bytes  RAM 
o  Sony  videodisc  player  1000  or  lOOOA 
o  12"  Sony  color  monitor 
o  Llghtpen  and  stereophonic  headset 

All  courseware  was  developed  using  MlcroTICCIT* s  ADAPT,  level  3,  authoring 
language.  In  addition,  students  at  each  station  were  provided  a  set  of  paper 
copy  topographic  maps  and  a  stereophonic  headset,  to  Insure  a  direct  link 
with  the  auditory  portions  of  the  Instruction  and  to  reduce  the  probability 
of  students  being  distracted  by  any  ambient  noise. 


Design  of  the  Evaluation 

The  independent  variable  In  this  evaluation  was  the  remedial  training  as 
presented  by  the  CVBI  courseware.  The  dependent  measure  was  the  overall  gain 
score  (posttest  minus  pretest)  for  each  of  the  four  remedial  tasks.  T-tests 
were  performed  for  both  paired  and  Independent  samples  as  appropriate 
(Jaccard,  Becker,  &  Wood,  1984). 

There  were  two  control  conditions  Included  to  address  the  alternative 
explanations  that  significant  Increases  on  posttest  measures  were  due  to 
soldiers'  Increased  familiarity,  gained  during  the  course  of  their  training 
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session,  trlth  the  test  Items  and  the  computer  delivery  system  (King  & 

Roblyer,  1984).  For  each  control  condition  participants  completed  pretests 
and  posttests  that  were  unrelated  to  the  training  they  received.  For  the 
first  control,  participants  In  the  remedial  group  completed  pretests  and 
posttests  on  two  land  navigation  tasks — Identify  Terrain  Features  and  Deter¬ 
mine  Elevation — unrelated  to  the  remedial  courseware.  In  the  second  control 
condition,  soldiers  from  the  land  navigation  group  completed  pretests  and 
posttests  on  two  remedial  tasks — Communicate  Using  Visual  Signals  and  Estab¬ 
lish  Tank  Firing  Positions — before  and  after  their  training  and  tests  on  land 
navigation. 

The  training  and  testing  sequence  for  both  the  remedial  and  land  naviga¬ 
tion  groups  Is  provided  In  Table  1.  Control  conditions,  In  which  pretests 
and  posttests  were  unrelated  to  the  training  received,  are  also  Indicated. 

The  training  and  tests  for  these  two  blocks  of  courseware,  remedial  and  land 
navigation,  were  Independent  with  no  overlapping  content.  But  the  tests  and 
training  for  each  block  of  courseware  were  presented  by  the  same  computer 
delivery  system. 


Table  1 

Training  and  Testing  Sequence  for  Remedial  and  Land  Navigation  Groups 


Sequence 

Group 

Pretest® 

Pretest 

Training 

Posttest 

Posttest® 

Remedial 

Land  Nav 

Remedial 

Remedial 

Remedial 

Land  Nav 

Land  Nav 

Remedial 

Land  Nav 

Land  Nav 

Land  Nav 

Remedial 

^Indicates  control  conditions  In  which  pretests  and  posttests  were  unre¬ 
lated  to  training  received. 


Instruments 


As  noted  above,  the  primary  Instrumentation  Involved  In  this  evaluation 
was  provided  by  the  MlcroTICCIT  host  computer  and  the  supporting  worksta¬ 
tions.  All  training  courseware  and  all  pretest  and  po^ttest  measures  for 
both  groups  were  on-line  products  supplied  directly  by  the  MlcroTICCIT  sys¬ 
tem. 


In  addition,  a  number  of  off-line  Instruments  were  also  administered  In¬ 
cluding  a  biographical  questionnaire,  a  courseware  evaluation  questionnaire, 
and  a  student  record  sheet.  The  biographical  questionnaire  Included  a  number 
of  Items  concerning  the  participant's  military  and  educational  background  and 
prior  computer  experience.  The  courseware  evaluation  questionnaire  contained 
two  5-polnt  Likert  scale  Items  that  asked  the  participants  to  rate  the  over¬ 
all  training  effectiveness  of  the  courseware,  and  ueveral  open-ended  Items 
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for  obtaining  any  recommendations  they  might  have  for  improving  the  course¬ 
ware.  Finally,  the  student  record  sheet,  which  was  maintained  by  the  class¬ 
room  proctors,  provided  a  data  sheet  for  recording  summary  scores,  time 
requirements  for  training  and  testing,  and  a  log  for  recording  system 
malfunctions,  student  problems  and  proctor  interventions.  Complete  copies  of 
each  of  these  off-line  instruments  are  provided  in  Appendix  A. 


Procedure 


Soldiers  were  trained  and  tested  individually  as  they  worked  at  their 
assigned  student  workstation.  Prior  to  the  training  session  each  participant 
received  a  brief  on-line  introduction  and  orientation  to  the  MicroTICCIT 
system.  This  introduction  explained  the  organizational  structure  of  the 
courseware  and  its  supporting  menu  and  prompt  messages.  In  addition,  this 
introduction  familiarized  students  with  the  use  of  the  lightpen  and  the 
avalllbllity  of  an  Advisor  function  to  assist  them  through  the  courseware. 

All  student  Inputs,  requests  and  answers  were  entered  by  using  a  lightpen. 
After  each  participant  completed  this  orientation,  pretests  were  administered 
for  each  task. 

Upon  completion  of  all  pretests,  participants  from  the  remedial  group 
began  their  training  on  the  remedial  courseware,  and  participants  from  the 
land  navigation  control  group,  on  the  land  navigation  courseware.  Any  par¬ 
ticipant  passing  the  pretest  for  a  given  task  was  excluded  from  the  training 
and  posttest  for  that  task.  This  exclusion  of  participants  not  requiring 
remediation  Is  consistent  with  19K  BNCOC's  procedures  for  diagnostic  testing, 
and  prevents  an  underestimation  of  the  courseware's  training  effectiveness 
due  to  celling  effects. 

While  all  participants  were  directed  to  complete  the  initial  objective 
and  instruction  components  for  each  of  the  tasks  trained,  their  training  cur¬ 
riculum  thereafter  was  determined  by  their  self-selections  and  computer-based 
prompts.  Instructional  prompts  based  on  each  soldier's  response  and  error 
patterns  were  provided  by  the  on-line  Advisor.  For  example,  a  student  making 
an  error  on  practice  problems  might  automatically  be  referred  by  the  Advisor 
to  a  related  help  component.  Or  after  successfully  answering  a  preset  number 
of  practice  problems,  the  student  would  then  be  further  advised  that  he  was 
probably  ready  to  take  the  appropriate  test. 

The  posttest  for  each  task  was  administered  immediately  after  the  train¬ 
ing  for  that  task  was  completed.  Participants  then  proceeded  through  train¬ 
ing  on  the  remaining  tasks,  maintaining  the  same  traln-te^t  sequence.  All 
participants  were  allowed  to  take  short  breaks  at  their  discretion,  but  they 
were  Instructed  not  to  discuss  the  training  and  test  material  until  after 
they  had  completed  all  courseware  and  posttest  measures. 

After  finishing  all  training  and  related  posttests,  participants  were 
administered  posttests  on  the  control  tasks  for  which  they  had  received  no 
training.  Finally,  all  participants  were  asked  to  complete  a  very  brief 
courseware  evaluation  questionnaire,  and  the  classroom  proctors  recorded  any 
extended  comments  or  recommendations  they  made  about  improving  the  courseware 
and  training  procedures. 
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Each  soldier's  session  lasted  approximately  three  hours  depending  upon 
the  participant's  pace  in  completing  the  training  and  testing  requirements. 
Throughout  the  entire  session,  classroom  proctors  were  present  to  assist 
students  with  any  problems,  input  proctor  functions  for  accessing  the  tests 
which  were  not  directly  accessible  to  students,  and  record  both  test  scores 
and  participants'  comments. 


Evaluation  Measures 


All  criterion  measures,  the  pretest  and  posttest  scores,  were  generated 
from  on-line  courseware  materials.  Tests  for  each  of  the  four  remedial  and 
two  land  navigation  tasks  consisted  primarily  of  multiple  choice  items.  Stu¬ 
dents  Indicated  their  answers  by  selecting  with  the  lightpen  their  choice 
from  among  those  response  alternatives  appearing  on  their  workstation  moni¬ 
tor.  Number  of  test  items  and  choices  per  item  varied  as  a  function  of 
courseware  content  and  design.  A  brief  description  of  each  test  is  provided 
below. 

Determine  Grid  Coordinates.  For  this  test  there  were  two  types  of 
evaluation  Items.  In  the  first,  students  determined  the  six-digit  coordi¬ 
nates  and  two-letter  100,000-square-meter  identifiers  for  points  indicated  on 
maps  displayed  on  the  workstation  monitor.  Answers  were  entered  using  the 
lightpen  and  a  numerical  keypad  displayed  in  one  comer  of  the  screen.  For 
the  second  set  of  items,  students  used  the  lightpen  to  move  a  cursor  across 
their  monitor-based  maps  to  the  location  of  the  coordinates  designated.  The 
test  consisted  of  five  items  from  each  type. 

Use  Visual  Signals.  For  this  test  various  hand-and-arm,  flag,  and  flash¬ 
light  signals  were  demonstrated  by  soldiers  in  field  settings  via  the  student 
monitor.  Students  were  then  required  to  either  select,  from  a  multi¬ 
ple-choice  list,  the  name  of  a  signal  demonstrated,  or  indicate  which  of  the 
four  signals  shown  was  the  one  named.  The  test  consisted  of  16  of  these 
items. 

Identify  Armored  Vehicles.  For  this  test  students  were  shown  pictures  of 
30  friendly  and  threat  armored  vehicles  as  viewed  from  various  perspectives. 
Students  were  required  to  identify  each  vehicle  as  friendly  or  threat  within 
10  seconds,  and  then  to  select  the  correct  nomenclature  within  an  additional 
10  seconds. 

Establish  Tank  Firing  Positions.  For  this  test  students  were  presented 
pictures  of  tanks  in  hull-dotm,  turret-down,  and  hide  positions,  as  well  as 
primary,  alternate,  and  supplementary  firing  positions.  The  test  consisted 
of  12  questions  for  which  students  were  required  to  correctly  Identify  posi¬ 
tions  depicted  or  "move**  their  tanks  into  positions  designated. 

Identify  Terrain  Features.  The  test  for  this  control  task  required  stu¬ 
dents  to  Identify  eight  major  terrain  features  indicated  on  various  maps  dis¬ 
played  on  the  screen.  Students  used  the  lightpen  to  either  pick  out  an 
example  of  a  designated  feature  on  a  map  or  identify  the  name  of  the  feature 
highlighted  by  computer  graphics.  The  test  consisted  of  12  items,  six  of 
each  type. 
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Determine  Elevation.  The  test  for  this  control  task  required  students  to 
determine  the  elevation  of  various  terrain  points  as  designated  on  their 
monitor-based  maps.  Answers  were  entered  using  the  llghtpen  and  the  numeri¬ 
cal  keypad  displayed  on  the  screen.  The  test  consisted  of  six  such  Items. 

Pretest  and  posttest  scores  were  obtained  by  assigning  one  point  to  each 
test  Item  correctly  answered  for  all  tasks  except  Identify  Friendly  and 
Threat  Armored  Vehicles.  For  this  vehicle  Identification  task  scoring  was  as 
follows:  each  correct  answer  scored  plus  two  points,  each  wrong  answer  minus 
two  points,  and  each  “don't  know"  answered  minus  one  point  (to  discourage 
guessing).  To  ensure  equivalent  measures.  Identical  Items  were  used  for  both 
the  pretest  and  posttests.  For  a  more  complete  description  of  the  test  Items 
the  reader  Is  referred  to  Knerr  et  al.  (1985). 


RESULTS  AND  DISCUSSION 


Tasks  Not  Trained 


As  previously  discussed,  the  research  design  for  this  evaluation  was  a 
pretest-posttest  design  In  which  the  effects  of  training  were  based  on  Im¬ 
provements  or  gains  In  students'  posttest  scores  over  and  above  their  pretest 
scores.  Higher  posttest  scores,  however,  may  have  been  due  to  students  In¬ 
creased  familiarity  with  the  computer-based  training  delivery  system  and/or 
the  test  Items  and  formats  rather  than  the  training  Itself.  To  partially 
control  for  these  potentially  confounding  effects,  each  participant  from  the 
remedial  group  completed  on-line  tests  for  two  nontralned  tasks  before  and 
after  their  work  with  the  remedial  courseware. 

A  summary  of  the  remedial  group's  control  data  on  tasks  for  which  they 
were  not  trained — Identify  Terrain  Features  and  Determine  Elevation — Is  pre¬ 
sented  In  the  upper  portion  of  Table  2.  The  average  gain  scores  for  these 
two  tasks,  .31  and  .31,  were  relatively  small.  Paired  comparison  t-tests 
revealed  a  modest  but  nonsignificant  Improvement  on  these  control  tasks, 

(38)  *  .93,  ^  >  .36  for  the  Identify  Features  task  and  ^  (38)  =  1.48, 

>  .14  for  the  Determine  Elevation  task. 

More  direct  evidence  for  ruling  out  the  effects  of  computer-experience 
and  test  familiarity  on  the  remedial  tests  per  se,  rather  than  the  unrelated 
land  navigation  tests,  was  provided  by  data  from  the  land  navigation  group 
which  Is  also  Included  In  the  upper  portion  of  Table  2.  As  stated,  the  land 
navigation  group  was  tested  but  not  trained  on  two  of  the  remedial 
tasks — Communicate  Using  Visual  Signals  and  Establish  Tank  Firing  Positions. 
Their  average  gain  scores  for  these  tasks,  .43  fo.^  Signals  and  .30  for  Posi¬ 
tions  again  showed  no  significant  Improvement  In  the  absence  of  training. 
Paired  comparison  t-test  values  were  ^  (43)  ••  1.61,  >  .11  for  the  Signals 

task  and  £  (42)  “  .96,  £  >  .33  for  the  Positions  task. 

In  conclusion,  the  modest  and  nonsignificant  gain  scores  for  the  four 
tasks  not  trained  provide  supporting  evidence  that  any  significant  Improve¬ 
ments  obtained  on  tasks  trained  were  probably  not  caused  by  familiarity  with 
the  test  Items  and/or  the  computer-based  delivery  system.  More  direct  evi¬ 
dence  Is  provided  In  the  following  section  where  these  modest  Improvements 
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due  to  familiarity  are  directly  compared  with  Improvements  caused  by  train¬ 
ing. 


Table  2 


Summary  of  Results  for  Tasks  Trained  and  Tasks  Not  Trained 


Tasks 

Number^  of 
Soldiers 

Number  of 
Items 

Pretest 

Average 

Posttest 

Average 

Gain 

Average 

Not  Trained 

Signals^ 

44 

16 

11.02 

11.46 

.43 

Positions® 

43 

12 

6.41 

6.63 

.30  . 

Features 

39 

12 

8.59 

8.90 

.31 

Elevation 

39 

6 

2.20 

2.51 

.31 

Trained 

Signals 

34 

16 

11.03 

14.59 

3.56 

Positions 

27 

12 

7.29 

11.48 

4.19 

Coordinates 

1  24 

10 

4.38 

7.96 

3.58 

Vehicles 

25 

24 

8.99 

15.9 

6.90 

Note.  Task  names  refer  In  order  to  the  following  tasks:  Communicate  Using 
Visual  Signals,  Establish  Tank  Firing  Positions,  Identify  Terrain 
Features,  Determine  Elevation,  Determine  Grid  Coordinates  and  Identify 
Friendly  and  Threat  Armored  Vehicles. 

^Indicates  control  tasks  from  land  navigation  group,  all  other  data  from 
remedial  group. 

^Numbers  of  soldiers  completing  both  pretests  and  posttest.  Original  pretest 
samples  were  n  *  44  for  land  navigation  group  and  n  >  39  for  remedial  group. 


Tasks  Trained 


Training  Effectiveness.  Summary  data  for  the  four  remedial  tasks  trained 
and  evaluated — Communicate  Using  Visual  Signals,  Establish  Tank  Firing  Posi¬ 
tions,  Determine  Grid  Coordinates  and  Identify  Friendly  and  Threat  Armored 
Vehicles — are  presented  In  the  lower  portion  of  Table  2.  As  noted  In  the 
Procedure  section,  participants  who  passed  the  pretest  for  any  task  were 
excluded  from  the  training  and  posttest  for  that  taik..  The  number  of  sol¬ 
diers  receiving  the  remedial  training,  therefore,  varied  by  task  as  Indicated 
In  the  first  column. 

For  the  remedial  tasks  trained.  Improvement  In  participants'  posttest 
performance  la  Indicated  by  the  average  gain  scores  presented  In  the  last 
column  of  Table  2.  Gain  scores  for  tasks  trained  ranged  from  3.56  for  the 
Signals  task  to  6.9  for  the  Vehicles  task,  relatively  large  gains  compared  to 
those  obtained  for  tasks  not  trained.  Paired  comparison  t-tests  of  pretest 
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versus  posttest  performance  for  each  of  the  four  remedial  tasks  revealed 
highly  significant  Increases  In  soldiers  posttest  scores,  all  £  values  less 
than  .001. 

A  more  direct  test  of  the  courseware's  training  effectiveness  Is  availa¬ 
ble  for  the  two  remedial  tasks — Communicate  Using  Visual  Signals  and  Estab¬ 
lish  Tank  Firing  Positions — on  which  the  land  navigation  group  was  tested, 
but  not  trained.  Between  groups,  remedial  versus  land  navigation,  t-test  com¬ 
parisons  of  these  gain  scores  were  highly  significant,  £  (76)  =  7.72, 

£  <  .001  for  the  Signals  task,  and  £  (68)  =  6.86,  £  <  .001  for  the  Positions 
task.  This  finding  of  significantly  greater  gains  for  the  remedial  group  on 
tasks  trained  compared  with  the  gains  of  the  land  navigation  group  on  these 
same  tasks  provides  the  most  direct  evidence  of  the  coursewares'  training  ef¬ 
fectiveness. 

A  more  practical  Indicator  of  this  courseware's  training  effectiveness 
may  be  a  more  traditional  measure  of  student  performance,  percentage  of  test 
Items  correctly  answered.  Table  3  provides  a  summary  of  the  percentage  of 
test  Items  correctly  answered  for  both  tasks  trained  and  tasks  not  trained. 
These  data  represent  test  scores  for  soldiers  who  failed  the  pretests  and 
thus  took  the  training  and  posttests.  The  gain  scores  for  tasks  trained, 
ranging  from  22%  to  36%  Improvement,  demonstrate  a  significant  training  ef¬ 
fect.  Gain  scores  for  tasks  not  trained,  ranging  from  2%  to  5%  Improvement, 
were  not  significant  and  again  dispel  the  rival  hypotheses  of  Increased  fa¬ 
miliarity. 


Table  3 

Average  Percentage  of  Test  Items  Correctly  Answered 


Tasks 

Pretest 

Posttest 

Gain 

Not  Trained 


Signals" 

69 

72 

3 

Positions® 

53 

55 

2 

Features 

72 

74 

2 

Elevation 

37 

42 

5 

Trained 

Signals 

69 

91 

22 

Positions 

61 

96 

35 

Coordinates 

44 

80 

36 

Vehicles 

36 

66 

30 

"indicates  data  from  land  navigation  group,  all  other  data  from  remedial 
group. 


In  the  case  of  a  normative  course  standard  (l.e..  Pass  >  70%),  the  reme¬ 
dial  courseware  raised  "class”  averages  for  the  Table  3  participants  from  "No 
Go"  to  "Go"  for  all  tasks  trained  except  Identify  Friendly  and  Threat  Armored 
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Vehicles.  Actual  course  standards  for  these  remedial  tasks  have  yet  to  be 
established  since  only  one  of  these  tasks.  Determine  Grid  Coordinates,  Is 
Included  In  19K  BNCOC's  current  POI.  Course  standards  are  an  Important  con¬ 
sideration  that  Is  beyond  the  scope  of  this  evaluation.  But  It  should  be 
noted  that  standards  for  these  remedial  tasks  for  personnel  entering  the  Army 
may  not  be  adequate  for  the  more  experienced  soldiers  typical  of  19K  BNCOC; 
and  conventional  test  measures  currently  used  for  entry  personnel  are  not 
always  directly  relatable  to  computer-based  testing. 

With  respect  to  the  Identify  Vehicles  task,  the  Identification  of  30 
friendly  and  threat  vehicles  Is  a  demanding  task  as  Indicated  by  the  partici¬ 
pants'  relatively  low  pretest  performance.  In  addition,  answers  were  scored 
correct  only  If  provided  within  10  seconds.  Nevertheless,  this  is  a  critical 
Ml  tank  commander  task  and  the  data  Indicate  that  some  participants  may  need 
to  be  retrained  and  retested  on  Vehicle  ID.  Another  alternative  Is  that  the 
on-line  Advisor  function  for  recommending  that  a  participant  is  probably 
ready  for  Vehicle  ID  testing,  based  on  percentage  of  practice  Items  answered 
correctly,  be  made  more  stringent. 

Other  data  related  to  training  effectiveness  were  obtained  from  the  par¬ 
ticipants'  subjective  evaluation  of  the  courseware  as  recorded  on  the  Post¬ 
test  Questionnaire.  On  the  average,  participants  rated  the  courseware's 
training  effectiveness  as  4.2  on  a  5-polnt  scale  (verbally  anchored  with 
"Good  Training"  for  a  4.0  rating  and  "Very  Good  Training"  for  a  5.0  rating). 
In  general,  90X  of  the  participants  rated  this  CVBI  training  as  either  "Good" 
or  "Very  Good,"  and  only  1  of  the  39  participants  rated  the  courseware  as 
below  average  training.  Another  evaluation  item  asked  participants  to  di¬ 
rectly  compare  the  effectiveness  of  the  CBI  courseware  with  conventional 
training  on  the  same  tasks.  Ninety  two  percent  of  the  participants  rated  the 
courseware  as  "Better"  or  "A  Lot  Better"  than  conventional  training,  and  only 
one  participant  rated  conventional  training  superior  to  the  computer-based 
Instruction. 

Participants  also  responded  to  a  number  of  open-ended  Items  on  the  Post¬ 
test  Questionnaire  that  asked  about  their  overall  reaction  to  the  CVBI 
courseware,  and  more  specifically  which  Instructional  features  were  most  and 
least  liked.  Over  half  of  the  participants  commented  favorably  on  the 
self-paced  nature  of  the  computer-based  training  and  the  option  to  review 
Instructional  segments  as  often  as  required.  Selected  comments  concerning 
the  option  to  review  Instructional  segments  were  "retraining  on  the  spot," 
"able  to  go  back  as  many  times  as  I  need,"  and  "great  to  train  where  I  got  a 
No  Go."  Concerning  the  courseware's  self-paced  nature  their  comments  ranged 
from  "your  own  pace  and  time"  and  "proceed  at  my  own  speed  and  not  some  fast 
talking  Instructor"  to  "skim  through  the  parts  I  was  familiar  with"  and 
"didn't  have  to  wait  for  others  to  catch  up."  Related  to  the  self-paced  and 
adaptive  nature  of  the  CVBI  courseware,  many  of  the  participants  characteriz¬ 
ed  this  computer-based  training  as  more  personal  than  conventional  training. 
"It  gave  you  what  you  needed,"  "the  graphics  and  instruction  were  directed  to 
me  and  not  just  a  class,"  "more  personal,  learned  at  my  pace"  and  "serious 
cs’«-on-one  competition."  Finally,  many  of  the  participants  commented  on  the 
clarity  and  simplicity  of  the  Instructional  materials  with  comments  such  as 
"simple  to  understand,  a  lot  simpler  to  remember,"  "the  computer  explained 
clearly,"  "more  explanatory  and  simple"  and  "broken  down  to  a  level  where  all 
soldiers  should  understand." 
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The  participants  also  provided  several  negative  comments  about  selected 
features  of  the  courseware.  Their  most  frequent  complaint  was  their  Inabil¬ 
ity  to  change  tests  answers.  The  experimenters  and  courseware  designers 
relearned  two  valuable  lessons  In  this  area.  First,  any  student  input  at  a 
workstation,  Including  a  test  answer,  may  be  the  result  of  an  accidental 
rather  than  Intentional  user  entry.  Students  must  have  the  opportunity  to 
both  verify  and  rectify  their  answers  immediately  after  they  have  been  en¬ 
tered.  The  second  lesson  Is  a  little  more  complicated.  How  long  should 
students  be  allowed  to  go  back  and  change  their  answers  on  a  test?  For  the 
relatively  open-classroom  we  had  anticipated  for  19K  BNCOC  in  which  students 
might  leave  their  workstation  at  any  time,  even  during  a  test,  the  software 
prevented  students  from  going  back  and  changing  their  answers  once  they  had 
moved  to  the  next  test  Item.  But  this  Inflexibility  is  not  typical  of  most 
test-taking  situations  In  which  no  answers  are  final  until  the  student  has 
"turned-ln"  the  completed  exam.  Before  Implementation  of  this  TTFA  course¬ 
ware  the  software  will  be  revised  to  provide  this  more  natural  test-taking 
scenario  in  which  answers  can  be  changed  until  students  have  completed  their 
exams,  but  proctor  and  security  measures  must  be  intensified. 

Training  Efficiency.  Another  Important  factor  In  the  evaluation  of  a 
training  technology,  or  development  of  a  courseware  product.  Is  training 
efficiency.  The  average  times  required  to  complete  training  and  testing  on 
this  CVBI  are  presented  In  Table  4.  On  the  average,  computer-based  pretests 
for  this  block  of  instruction  were  completed  In  less  than  one  hour  (i.e.,  47 
minutes).  For  participants  not  passing  these  pretests  and  requiring  remedial 
training,  the  average  time  required  to  complete  their  training  and  the  post¬ 
tests  was  less  than  three  hours. 


Table  4 

Completion  Times®  for  Training  and  Testing 


Tasks 

Pretests 

Training  + 

Posttest 

Total 

Average 

Range 

Average 

Range 

Average 

Range 

Signals 

10 

4-17 

43 

16-  99 

53 

20-116 

Positions 

5 

3-11 

15 

10-  23 

20 

13-  34 

Coordinates 

17 

6-31 

41 

16-  99 

58 

22-130 

Vehicle 

15 

2-24 

73 

14-139 

88 

16-163 

47 

172 

56-360 

219 

71-443 

®Completion  times  expressed  In  minutes. 


Determining  the  relative  training  efficiency  of  this  CVBI  compared  to 
conventional  training  is  difficult  because  remedial  training  Is  not  a  formal 
component  In  19K  BNCOC's  Program  of  Instruction.  While  diagnostic  testing  Is 
routinely  conducted  for  this  course,  remedial  training  is  primarily  each 
students  responsibility.  Evening  study  hall  times  for  remediation  are  sched¬ 
uled  into  the  19K  BNCOC  course  cycle  and  proctors  are  available  to  assist 
students,  but  official  records  of  remedial  training  times  are  not  maintained. 
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Training  time  for  each  of  these  Skill  Level  1  remedial  tasks  are  available 
for  entry-level  personnel,  but  a  comparison  of  these  entry-level  data  with 
the  experienced  personnel  in  this  setting  would  be  misleading;  remedial 
training  for  the  current  participants,  and  19K  BNCOC  personnel  in  general,  is 
a  review  of  the  entry-level  training  they  received  approximately  three  years 
earlier. 

Nevertheless,  the  automated  availability  of  training  efficiency  data, 
such  as  that  provided  in  Table  4,  is  an  important  feature  unique  to  com¬ 
puter-based  training  technologies.  Adoption  of  this  and  similar  CBI 
Army-wide,  would  readily  provide  Army  trainers  a  valid  indicator  of  skill 
level  retention  and  cadre  requirements  for  remediation. 

Finally,  the  range  of  times  required  for  training  and  testing  that  are 
presented  in  Table  4  provide  an  indication  of  the  diversity  of  soldier  train¬ 
ing  requirements  within  a  relatively  homogenous  sample.  On  the  average,  the 
total  time  required  to  complete  all  training  and  testing  for  these  remedial 
tasks,  ranged  from  a  minimum  of  just  over  one  hour  to  a  maximum  requirement 
well  over  seven  hours.  Siace  remediation  is  ultimately  each  soldier's  re¬ 
sponsibility,  the  availability  of  self-paced  and  adaptive  instructional  tech¬ 
nologies  is  an  important  training  resource.  Results  such  as  those  provided  in 
Table  4  should  make  both  instructors  and  students  of  19K  BNCOC  more  aware  of 
each  soldiers  unique  remediation  requirements,  and  provide  administrators  the 
data  needed  to  more  effectively  schedule  both  formal  and  remedial  training 
within  the  19K  BNCOC  course  cycle. 


CONCLUSIONS 


Results  from  this  evaluation  suggest  CVBI  courseware  provides  an  effec¬ 
tive  training  technology  for  meeting  at  least  some  of  the  remedial  training 
requirements  of  19K  BNCOC.  For  each  of  the  remedial  tasks  evaluated — Use 
Visual  Signals,  Establish  Tank  Firing  Positions,  Determine  Grid  Coordinates 
and  Identify  Armored  Vehicles — participants'  gain  scores  demonstrated  sig¬ 
nificant  improvements.  The  design  of  the  evaluation  ensured  that  these  im¬ 
provements  were  not  due  to  participant's  increased  familiarity  with  the  test 
items  and  formats  or  the  computer  delivery  system.  With  respect  to  normative 
course  standards,  the  courseware  raised  participants  from  a  pretest  average 
of  ”No  Go”  to  a  posttest  average  of  "Go”  after  one  training  session  for  three 
of  the  four  tasks  evaluated.  In  addition,  participants'  subjective  evalua¬ 
tions  of  this  courseware  suggest  that  they  regarded  it  as  better  than  conven¬ 
tional  training  methods.  While  training  efficiency  could  not  be  directly 
determined,  the  diversity  of  the  participants'  need  for  remedial  training 
strongly  supports  the  adaptive  nature  of  training  technologies  such  as  CVBI. 

This  remedial  courseware  is  now  ready  for  validation  trials  in  19K  BNCOC. 
Key  characteristics  of  the  validation  trials  are  that  students  currently 
enrolled  in  the  BNCOC  course  will  receive  this  CVBI  training  and  testing  and 
course  instructors,  rather  than  TTFA  support  personnel,  will  operate  the 
courseware  delivery  system.  Validation  trials  are  expected  to  be  followed  by 
evaluations  of  the  courseware's  effectiveness  in  transfers  to  field  settings. 
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APPENDIX  A 

OFF-LINE  EVALUATION  MEASURES 


BIOGRAPHICAL  QUESTIONNAIRE 


Subject  # 

Date 

Social  Security  # 

Rank  E  - 

_  0  ' 

■1 .  How  long  have  you  been  in  the  service? 

years 

months 

2.  How  old  are  you?  _  years  _  months 

3.  What  is  your  primary  MOS/SC?  _ 

What  is  your  secondary  MOS/SC?  _ 

5.  What  is  your  current  Unit? 

6.  Please  check' each  of  the  following  courses  you  have  attended: 

PLDC  _  BNCOC  _  ANCOC  _  AOC  _  AOAC  _ 

7.  Check  each  of  the  following  statements  that  are  true  for  you. 

_  I  have  no  previous  experience  with  computers. 

_  I  frequently  play  video  games. 

_  I  own  a  home  computer. 

_  I  have  received  formal  training  on  how  to  use  a  computer. 

_  I  have  previously  participated  in  training  delivered  by  a  computer. 

_  I  have  a  lot  of  experience  with  computers. 

8.  What  is  your  highest  academic  level  completed?  (Check  one  below.) 

High  School  _  GED  _  Some  College  College  Graduate 

9.  How  would  you  describe  your  "handedness"  or  preferred  hand?  (Check  one 
below. ) 

Right  handed  _  Left  handed  _  Both  hands 


PT  5682 
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STUDENT  RECORD  SHEET  FOR  REMEDIAL  TRIALS 
(EVEN  numbered  students) 


Student  Number;  Primary _ Additional _ 

PRETEST  SCORE  POSTTEST  SCORE 

SEQUENCE  G  OR  NG/#  CORRECT  G  OR  NG/#  CORRECT  PROBLEH/SOLUTION 


Land  Navigation 

ID  Features  (12) 

/ 

/ 

Det.  Elevation  (  6) 

/ 

/ 

Remedial 


Six  Digit  Grid  (10)  _ 

/ 

/ 

/ 

Radio  Set  (  3)  _ 

/ 

/ 

/ 

Visual  Signals  (16)  _ 

/ 

/ 

/ 

Tank  Positions  (12)  _ 

/ 

/ 

1 

/ 

/ 

/ 

/ 

Land  Navigation 

ID  Features  (12) 

/ 

1 

Det.  Elevation  (  6) 

/ 

/ 

PT  5682 
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POSTTEST  QUESTIONNAIRE 


Student  #: 


Date: 


We  would  like  to  ask  you  a  few  final  questions  about  how  you  evaluate  the 
computer-based  training  that  you  have  just  experienced. 

1.  In  general  what  did  you  like  best  about  the  computer-based  training? 
Please  briefly  write  your  answer  below. 


2.  In  general  what  did  you  like  least  about  the  computer-based  training? 
Please  briefly  write  your  answer  below. 


3.  How  do  you  rate  the  overall  effectiveness  of  this  computer-based  training? 
Please  circle. the  number  below  (1-5)  that  indicates  your  answer. 


VERY  POOR 
TRAINING 


POOR 

TRAINING 


AVERAGE 

TRAINING 


GOOD 

TRAINING 


VERY  GOOD 
TRAINING 


4.  How  do  you  rate  this  computer-based  training  compared  to  standard  or 
conventional  training  in  the  same  subject  areas?  Please  circle  the  number 
below  (1-5)  that  indicates  your  answer. 

Computer-based  training  is: 


MUCH  WORSE  A  LITTLE  THE  SAME  A  LITTLE  A  LOT  BETTER 

THAN  WORSE  THAN  AS  BETTER  THAN  THAN 

CONVENTIONAL  CONVENTIONAL  CONVENTIONAL  CONVENTIONAL  CONVENTIONAL 


5.  What  could  be  done  to  improve  the  computer-based  training?  Please  write 
suggestions  below. 
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